1. Field of the Invention
The present invention generally relates to data signal detection, compression and storage and, more particularly, to pre-detection and possible detection of weak signals in the presence of noise, partial removal of noise and compression of signals for storage consistent with detected data signal preservation.
2. Description of the Prior Art
Many applications exist at the present time and are foreseeable in which vast quantities of signals are acquired and which may exceed the capacity of facilities for processing the signals concurrently with the acquisition thereof. Such applications include, but are not limited to astronomical observations (e.g. optical and radio telescope images), surveillance video and other sensing arrangements such as intrusion monitoring, meteorological observations, manufacturing facility condition monitoring, communications monitoring, medical monitoring and/or imaging, radar and sonar observations, location tracking (e.g. through a Global Positioning System), and the like. As can be appreciated, in many of these applications, the nature of the data signal of interest may not be known with precision prior to implementation of the application but where logging of as much data as possible may be desirable in order to later search for data which may correspond to an anomalous observation such as determining the cause of a variation of manufacturing yield, determination of an unanticipated failure mode, determining the orbit of an astronomical object from data prior to its discovery (e.g. data in which the object might have been observable but was not detected), and the like. Similarly, many such applications produce signals in which high noise levels are unavoidable and may substantially exceed the levels of signals which, if found, may later be determined to be of interest.
Both the extraction of data signals from noise and the extraction of data of interest from other data in a signal may require specialized and computationally intensive processing, the nature of which may vary widely and require extensive experimentation to determine the nature of the data signal of interest as well as to reduce other signals which are not of interest and are thus effectively noise. Such experimentation and/or processing may require extended periods of time and may be performed long after the acquisition of the signal. Therefore, it is desirable in such applications to store signals with as much fidelity as can be practically accommodated in order to avoid loss or diminution or obscuring of data signals which may later be found to be of interest even though such vast amounts of data, as a practical matter, cannot be accommodated without some degree of compression which will necessarily result in some degree of data signal degradation or inclusion of some degree of compression noise, even though compression techniques exist (e.g. JPEG, MPEG, MP3, etc.—it is assumed that particular compression techniques are known and preferred for particular applications) which can provide an arbitrary degree of compression and optimal degree of fidelity to the original signal for a given amount of compressed data which can, in fact, be accommodated.
Lossless compression techniques are also known but lossless data compression of a signal in the presence of noise generally achieves only about a two to one compression. True noise is incompressible. Further, noise disrupts the predictability of any data signal which may exist within the noise and therefore interferes with any compression algorithm which may be applied. Also, it has been attempted to simulate and compress a noisy signal by separating the data signal from the noise and representing the noise by the parameters thereof in order to achieve a greater degree of compression of a noisy signal. However, this approach is processing intensive to separate the data from the noise while the resulting compressed signal, when decompressed, merely imposes a texture on the extracted signal and does not preserve any data signal which may be hidden in the noise (e.g. a low level data signal in addition to a larger data signal which can, in fact, be separated from noise).
However, in applications such as those alluded to above, it is often the case that data signals may not be present for substantial periods of time and the signal obtained may comprise only noise while the noise may effectively prevent or greatly impede the determination of whether or not a data signal is also present within the noise; thus, if it is desired to store the signal, presenting the problem of requiring storage of vast amounts of incompressible noise. (As used herein, the term “data signal” will denote a signal other than noise which may contain information or otherwise be found to be of interest whereas the term “signal”, alone, will denote a signal which includes noise and may or may not include a “data signal”.) On the other hand, if no data signal is present in the signal acquired there is no need to store or further process the signal and the signals where no data signal is present can be discarded or the noise function can be represented with an arbitrary representation of the noise function which represents a noise compression by an arbitrary amount but would not preserve the actual noise or any small signal obscured within it. However, no techniques exist which can provide such a determination with sufficient rapidity to reliably avoid storage of seemingly “noise only” signals where a data signal may be of lower amplitude or power than the noise component or to allow for storage or compression without actually determining the signal of interest. On the contrary, numerous theories have been substantially accepted in the art that such discrimination or screening of signals for data signals cannot be accomplished below a certain signal-to-noise ratio (SNR) thresholds and processing to improve the SNR often requires not only knowledge of the presence of a data signal but some characteristics of the data signal in order to preferentially enhance it. At the present state of the art, therefore, where no characteristics of a data signal are known in advance, the data signal must have a threshold greater than that of the noise to be detected. For most automated data signal detection, a SNR of +10 db is commonly considered to be the minimum signal level which permits detection of unknown data signals amid noise for purposes of screening acquired signals for the presence of data signals having an arbitrary form which is not known in advance.